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Appendix A. Overview of the Building Culturally Responsive Systems professional 
development program on culturally responsive practices 


In 2009 Andreal Davis developed a professional development program on culturally responsive practices called “7 
Experiences” for the Madison Metropolitan School District. Its goal was to build educators’ skills in implementing 
culturally responsive practices in their classrooms. The program was initially offered to four schools in the district 
(Davis & Lehman, 2013). In 2013/14 the Wisconsin Response to Intervention Center, which assists Wisconsin 
schools with building capacity and adopting and implementing high-quality education practices, started offering 
the program statewide. In addition, the Wisconsin Response to Intervention Center changed the name of the 
program to Building Culturally Responsive Systems, encouraged schools to complete the program, and 
recommended that schools implement culturally responsive practices within their multi-tiered systems of 
supports. Although the program has been offered statewide for six years, little research has been conducted on 
its reach across the state or its relationship to student or school outcomes. 


The statewide program is a series of five in-person sessions designed for school and district teams. It is intended 
to help educators understand the policies and practices related to inequitable student outcomes and develop 
their ability to “create an inclusive learning environment and develop practices, strategies, and curriculum that 
include and honor the life experiences and cultures of their students” (Wisconsin Response to Intervention Center, 
2019). The program achieves this goal by developing cultural competence, exploring power and privilege, and 
examining subtleties of culture and diversity. Sessions include guided self-reflection, discussion of examples of 
culturally responsive practices (for example, incorporating literacy books that match students’ demographic 
characteristics, using call-and-response techniques, and administering learning environment surveys), and 
discussions on how to use data to understand and address specific equity issues. 


The five sessions run from 8:30 a.m. to 5:30 p.m. each day. The program is offered in four locations across the 
state. For example, in the 2017/18 school year the program was offered in Eau Claire, Madison, Oshkosh, and 
Tomah. The dates for each session are spread across the school year. For example, at the Madison location, 
sessions were held on November 7, November 8, April 17, April 18, and May 15. The program costs $325 per 
participant, but the cost is waived for districts with disproportionate representation of racial/ethnic minority 
groups in special education (Wisconsin Response to Intervention Center, 2019). 
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Appendix B. Methods 


This appendix includes detailed information about the data used for this report as well as the methods used to 
complete the analyses. 


Data sources 

The study used data provided by the Wisconsin Response to Intervention Center and the Wisconsin Department 
of Public Instruction as well as publicly available data from the U.S. Department of Education’s Common Core of 
Data (table B1). 


Research question 1. To address research question 1 on participation in the Building Culturally Responsive Systems 
professional development program, the study team used data from the Wisconsin Response to Intervention 
Center on the dates during which teachers and administrators from each school participated in the program. 


Research question 2. To address research question 2 on the differences in characteristics of schools that 
participated in the program and of schools that did not participate, the study team used the same data on 
participation from the Wisconsin Response to Intervention Center that were used to address research question 1 
as well as data on school characteristics from the Wisconsin Department of Public Instruction and the U.S. 
Department of Education’s Common Core of Data. 


The data from the Wisconsin Department of Public Instruction included school enrollment, percentage of students 
by race/ethnicity (American Indian, Asian, Black, Hispanic, Pacific Islander, two or more races/ethnicities, 
unknown race/ethnicity, and White), percentage of students eligible for the national school lunch program, and 
the district in which each school was located. 


The data from the U.S. Department of Education included school Title | status, number of teachers, teacher-to- 
student ratio, locale (city, suburb, town, or rural), and grade span. 


Research question 3. To address research question 3 on school implementation of culturally responsive practices 
in reading instruction and math instruction within their multi-tiered systems of supports (MTSS), the study team 
used data from the School-Wide Implementation Review surveys administered by the Wisconsin Response to 
Intervention Center. Schools responded to 10 items on two surveys on school practices within MTSS, one on 
practices in reading instruction and one on practices in math instruction. Responses were on a five-point scale: 
not in place, purpose building, infrastructure, initial implementation, and full implementation. The Wisconsin 
Response to Intervention Center converted the responses to a numerical scale of 0-4. 


Data for addressing research question 4. To address research question 4 on the relationship between program 
participation school-level academic and behavior outcomes, the study team used the same data on participation 
from the Wisconsin Response to Intervention Center that were used to address research question 1 as well as 
data on school-level academic and behavior measures (mean scale scores on English language arts and math tests; 
closing gaps scores for English language arts, math, and high school graduation rate;? attendance rate; suspension 
rate; and expulsion rate) from the Wisconsin Department of Public Instruction. 


For students in elementary and middle school, the study included mean scale scores from the Wisconsin 
Knowledge and Concepts Exam (administered in 2013/14 and years prior), the Badger Exam (administered in 
2014/15), and the Forward Exam (administered in 2015/16—2018/19). For students in high school, the study 
included scores from ACT assessments. The Wisconsin Department of Public Instruction standardized test scores 
by test, grade, and school to allow for easy comparisons. 


1 Closing gaps scores for high school graduation rate were excluded from the analysis due to a high level of missing data. 
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The closing gaps score measure a school’s progress toward closing achievement gaps between target groups 
(racial/ethnic minority students, students with disabilities, students with limited English proficiency, and students 
who are economically disadvantaged) and comparison groups (White students, students without disabilities, 
students classified as fully English proficient, and students who are not economically disadvantaged). Scores are 
calculated on a scale from 1 to 100 as part of the school’s annual report card. Higher scores indicate that the gap 
between groups is narrowing, and lower scores indicate that the gap is widening. Scores increase when the 
performance of students in target groups increases but the performance of students in comparison groups 
remains the same as well as when the performance of students in target groups remains the same but the 
performance of students in comparison groups declines. 
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Table B1. Summary of data used to answer research questions 


Data element 


Years of data used 


in analysis Source 


Research question 1. What percentage of schools participated in the program? 


Dates during which teachers and administrators from each 
school participated in the professional development program 
on culturally responsive practices 


2013/14-2018/19 Wisconsin Response to Intervention 
Center (obtained through a data 


request) 


Research question 2. How are the characteristics of the schools that participated in the program different from the 


characteristics of the schools that did not participate? * 
School enrollment 


School-level racial/ethnic makeup (American Indian, Asian, 
Black, Hispanic, Pacific Islander, two or more races/ethnicities, 
unknown race/ethnicity, and White) 


School-level percentage of students eligible for the national 
school lunch program 


School district in which schools were located 
Title | status 

Number of teachers 

Teacher-to-student ratio 

School locale 


School grade span 


2012/13-2017/18 
2012/13-2017/18 


Wisconsin Department of Public 
Instruction (obtained through a data 
request) 


2012/13-2017/18 


2014/15-2017/18 
2012/13-2016/17 
2012/13-2016/17 
2012/13-2016/17 
2012/13-2016/17 
2012/13-2016/17 


U.S. Department of Education’s 
(2018) Common Core of Data 


Research question 3. What percentage of schools that participated in the program and what percentage of schools that did 
not participate reported implementing culturally responsive practices within their MTSS? 


Responses to survey on implementation of culturally 
responsive practices in reading instruction 


Responses to survey on implementation of culturally 
responsive practices in math instruction 


2013/14-2018/19 Wisconsin Response to Intervention 
Center (obtained through a data 


2013/14-2018/19  Tequest) 


Research question 4. Is program participation related to school-level academic and behavior outcomes, after school 
characteristics and pre-program program academic and behavior measures are accounted for? 


Mean English language arts test scores 
Mean math test scores 

Closing gaps score for English language arts 
Closing gaps score for math 

Attendance rate 

Suspension rate 


Expulsion rate 


2012/13-2017/18 
2012/13-2017/18 
2012/13-2017/18 
2012/13-2017/18 
2012/13-2017/18 
2012/13-2017/18 
2012/13-2017/18 


Wisconsin Department of Public 
Instruction (obtained through a data 
request) 


a. All data elements used to address research question 1 were also used to address research questions 2, 3, and 4. 
Source: Authors’ compilation. 


Study sample 
The analytic sample included 2,290 public schools in Wisconsin (1,297 elementary schools, 72 combined 
elementary/secondary schools, 390 middle schools, and 531 high schools). 


2 The study used school-level data because student-level data were not available. 
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Data preparation 


Data from the Wisconsin Response to Intervention Center were merged with data from the Wisconsin Department 
of Public Instruction using a school code variable in both datasets. All records were kept except for any duplicate 
rows. The study team requested National Center for Education Statistics school identification codes from the 
Wisconsin Department of Education and used them to merge data from the U.S. Department of Education’s 
Common Core of Data with data from the Wisconsin Response to Intervention Center and the Wisconsin 
Department of Public Instruction. The study team dropped records that did not have the identification codes 
needed to merge the datasets. The study team also dropped 66 schools with only one or fewer teachers on 
average (including many nontraditional schools such as virtual schools and education centers) and 1 school with 
an average attendance rate of 0. The final data file had 2,290 school records. 


Program participation. The study team conducted the following manipulations and calculations to prepare the 
data on program participation for analysis: 


e A binary variable was created to indicate whether a school participated in the program based on the dates 
during which teachers or administrators from the school participated in the program. The study team assigned 
a code of 1 to schools from which teachers or administrators participated in at least one of the program’s five 
sessions in any year and a code of 0 to schools from which no teachers or administrators participated in the 
program in any year. 


e An additional variable was created to indicate the year in which a school sent teachers or administrators to 
the program. This variable was used in calculating pre-program scores and outcomes one year, two years, and 
three years after the program. Because schools participated in the program across multiple years, it was not 
possible to use a single year as the “pre-program” year for schools that did not participate. To establish a 
reference year for the calculation of pre-program values and post-program outcomes for schools that did not 
participate in the program, the study team randomly assigned these schools to a program year in percentages 
proportional to the percentages of schools that did participate in each year (table B2). The random assignment 
involved three steps: assigning a random number to schools that did not participate in the program, ordering 
those schools by that number, and distributing those schools across intervention years in the same 
proportions as schools that did participate. 


Table B2. Number and percentage of schools that participated in the program by year, 2013/14—-2018/19 


Number of schools that Percentage of schools that 

participated in the program participated in the program 
2013/14 24 25.0 
2014/15 33 34.4 
2015/16 11 11.5 
2016/17 2 2.1 
2017/18 22 22.9 
2018/19 4 4.2 


Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center and the Wisconsin Department of Public Instruction. 


School-level characteristics. The study team conducted the following manipulations and calculations to prepare 
the data on school-level characteristics for analysis: 


e School enrollment; the percentages of Black students, Hispanic students, and White students; the percentage 
of students who were eligible for the national school lunch program; the number of teachers; and the teacher- 
to-student ratio were each averaged across all years of data for each school. 
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American Indian students, Asian students, Pacific Islander students, students who identify as two or more 
races/ethnicities, and students with unknown race/ethnicity were combined into a single category, students 
of other race/ethnicity, and the percentage of students in that category was averaged across all years of data 
for each school. 


Title | status was recoded as a binary variable based on the most recent Title | status was used for each school. 
The original Common Core of Data Title | status variable had six categories. The study team assigned a code 
of 1 to schools that were eligible for Title | Targeted Assistance and received the assistance, schools that were 
eligible for Title | Targeted Assistance and did not receive assistance, schools that were eligible for the Title | 
Schoolwide Program and received the program, and schools that were eligible for the Title | Schoolwide 
Program and did not receive the program. The study team assigned a code of 0 to schools that were not 
eligible for either Title | Targeted Assistance or Schoolwide Program. 


The study team used the most frequent urban-centric locale designation (city, suburb, town, or rural) across 
all years of the study for each school. 


A binary variable was created to indicate whether a school was located in one of the “Big Five” school districts 
(Milwaukee, Madison, Racine, Kenosha, and Green Bay). The “Big Five” school districts are the five largest 
districts in Wisconsin and serve the majority of Black students in the state. School codes that are assigned by 
the Wisconsin Department of Public Instruction were used to create the binary variable. The study team 
assigned a code of 1 to schools located in one of the “Big Five” districts and a code of 0 to schools located in 
other districts. 


Implementation of culturally responsive practices in reading and math instruction within MTSS. The study team 
conducted the following manipulations and calculations to prepare the data on implementation of culturally 
responsive practices for analysis: 


To ensure the 10 items were reliable, the study team conducted a series of internal consistency checks. The 
10 items had a Cronbach’s alpha of .95 for reading and .96 for math. Because of the high internal consistency 
of the 10 items, the study team used the 10 items in determining whether schools were implementing 
culturally responsive practices. 


Binary variables were created to indicate whether a school implemented culturally responsive practices in 
reading instruction and in math instruction. The variables were based on the mean score for the 10 survey 
items across all years. The study team assigned a code of 1 to schools with a mean score of 3 or higher and a 
code of 0 to schools with a mean score of less than 3. A threshold of 3 was chosen because it indicated that a 
school was implementing some culturally responsive practices. 


School-level academic and behavior measures. The study team conducted the following manipulations and 
calculations to prepare the data on school-level outcomes for analysis: 


For each outcome the average of all values prior to the year in which a school participated in the program was 
used as the pre-program academic and behavior measures. For example, if a school participated in the 
program in 2017/18, the mean attendance rate from 2012/13 to 2016/17 was used as the pre-program 
attendance score. 


Values for each outcome one year, two years, and three years after participation were calculated. For 
example, for a school that participated in the program in 2012/13, the one-year post-program attendance 
rate was from 2013/14, the two-year post-program attendance rate was from 2014/15, and the three-year 
post-program attendance rate was from 2015/16. If data were not available for one year, two years, or three 
years after participation, the data were considered missing. 
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e Attendance rate. The average pre-program attendance rate and the average attendance rate one year, two 
years, and three years after participation were calculated for each school. 


e Mean English language arts and math test scores. The average pre-program English language arts and math 
standardized test z scores and the average English language arts and math standardized test z scores one year, 
two years, and three years after participation were calculated for each school. 


e Closing gaps scores for English language arts and math. Average pre-program closing gaps scores for English 
language arts and math and the average closing gaps scores one year, two years, and three years after 
participation were calculated for each school. Closing gaps scores for graduation rate were excluded from the 
analysis due to a high level of missing data (see next section). 


e Suspension rate. The average pre-program suspension rate and the average suspension rate one year, two 
years, and three years after participation were calculated for each school. 


e Expulsion rate. The average pre-program expulsion rate and the average expulsion rate one year, two years, 
and three years after participation were calculated for each school. 


Missing data 

Because the rate of missing data was low for the majority of the variables (under 7.2 percent; table B3), the study 
team used listwise deletion to handle missing data.? However, several variables had high levels of missing data, 
which affected the analysis for research question 4. Specifically, implementation of culturally responsive practices 
was removed from the regression analysis and included only in the descriptive analysis, and closing gaps scores 
for graduation rate were removed from both the descriptive analysis and the regression analysis. 


3 Using listwise deletion, a school was excluded from analyses if it was missing a value for a variable used in analysis. 
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Table B3. Number of valid cases and percentage of missing cases for each variable 


NWETar]o)(-1 Number of valid cases Percentage of missing cases 
Program participation 


Participation in the professional development program on 2,290 0.0 
culturally responsive practices 


Implementation of culturally responsive practices 
In reading instruction? 866 62.2 
In math instruction? 596 74.0 


School characteristics 


Average school enrollment 2,287 1.3 
Percentage of White students 2,287 1.3 
Percentage of Black students 2,287 1.3 
Percentage of students of “other race/ethnicity” 2,287 1.3 
Percentage of Hispanic students 2,287 1.3 
Percentage students eligible for the national school lunch program 2,287 1.3 
Title | status 1,616 5.3 
Number of teachers 2,246 1.9 
Teacher-to-student ratio 2,246 1.9 
School locale 2,277 0.6 
School grade span 2,290 0.0 
School-level outcomes 

Mean English language arts test scores 2,125 7.2 
Mean math test scores 2,125 7.2 
Closing gaps score for English language arts 1,784 22.1 
Closing gaps score for math 1,784 22.1 
Closing gaps score for graduation rate? 262 56.6 
Attendance rate 2,287 1.3 
Suspension rate 2,287 1.3 
Expulsion rate 2,287 1.3 


a. Removed from the regression analysis because of the high level of missing data. A nonresponse analysis indicated that whether a school was missing data 
on this variable was meaningfully related to school characteristics. Therefore, any findings related to this variable cannot be generalized to the population 
of schools in Wisconsin. 

b. Excluded from all analyses because of the high level of missing data. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center, the Wisconsin Department of Public Instruction, and the U.S. 
Department of Education’s Common Core of Data. 


Analysis 
To address research question 1, the study team used the data on dates of school participation to calculate the 
number and percentage of schools that participated in the program. 


To address research question 2, the study team compared the average school-level characteristics of schools that 
participated in the program and of schools that did not participate in the program. Group differences of 5 
percentage points or more were considered meaningful. 
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To address research question 3, the study team used the data from the School-wide Implementation Review 
surveys to calculate the number and percentage of schools that reported implementing culturally responsive 
practices within their MTSS by whether they had participated in the program. 


To answer research question 4, the study team conducted two sets of analyses. First, the study team compared 
average school-level academic and behavior outcomes for schools that participated in the program and schools 
that did not participate. Second, the study team used a series of regression models to examine the relationship 
between program participation and those outcomes. The regression models used the average school 
characteristics of all schools; participation in the program during any year; and school outcomes (all continuous 
outcomes) for one year, two years, and three years after the program. Because these analyses used the population 
of schools in Wisconsin, the study team did not conduct tests for statistical significance. To identify meaningful 
differences, the study team calculated effect sizes and reported standardized coefficients. Effect sizes and 
standardized coefficients with an absolute value greater than .15 were considered meaningful. 


Reference 


U.S. Department of Education, National Center for Education Statistics. (2018). Common Core of Data. The Local Education 
Agency (School District) Universe Survey, 2013/14-2016/17. Retrieved January 14, 2020, from 


https://nces.ed.gov/ccd/pubagency.asp. 
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Appendix C. Supporting analyses 


This appendix presents additional findings for research question 2 on differences in the characteristics of schools 
that participated in the Building Culturally Responsive Systems professional development program and of schools 
that did not participate and for research question 4 on the relationship between program participation and school 
outcomes. 


Comparing school characteristics 

The analysis for research question 2 included a breakdown of the locations across the state for schools that 
participated in the program (map C1) as well as school characteristics by school grade span (elementary school, 
elementary/secondary school, middle school, and high school; table C1). Only 6 of the 96 schools that participated 
in the program were located in one of the “Big Five” school districts, which serve the majority of Black students in 
the state. Of those six schools, four were elementary schools and two were middle schools. Schools that 
participated in the program were more often located in cities and suburbs than schools that did not participate. 
Of schools that participated in the program, 42 percent were located in cities, and 30 percent were located in 
suburbs. Of schools that participated in the program, middle schools were more likely to be located in cities than 
in other areas, and high schools were more likely to be located in suburbs than in other areas. Of schools that did 
not participate in the program, middle schools and high schools were more likely to be located in rural areas than 
in other areas. Of schools that did not participate in the program, elementary/secondary schools had the highest 
percentage of Black students (36 percent). No elementary/secondary schools participated in the program. 
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Map C1. Schools that participated in the Building Culturally Responsive Systems professional development 
program in Wisconsin were clustered around the big cities across the state 


Eau Claire 


WISCONSIN 


rt Lea Austin 


Cedar Rapids Sources: Esri, HERE Garmin, FAO, NOAA USGS, © OpenStreetMap contributors, and the GIS U 


Note: The sample consisted of the 96 schools that participated in the program. Larger circles indicate that more schools in a school district participated. 
Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center. 


Table C1. Characteristics of Wisconsin schools, by participation in the professional development program on culturally responsive practices 
and school grade span, 2013/14-—2018/19 


Schools that participated in the program Ro Koto) CM ant-imel(oMarelmey-lad(aloy-1c-M lam dalcM o)cey-4e-1 00) 
: Elementary/ 
Elementary Middle High Elementary secondary Middle 
(o)V-T¢-11| schools schools schools (o)V(-T¢-11| schools schools schools High schools 

Characteristic (n = 96) (n = 56) (n = 20) (n = 20) (n=2,194) (n= 1,241) (n =72) EEY,0)) (n=511) 
Average school enrollment 582.3 384.2 616.4 1103.2 378.4 341.6 292.5 390.4 471.3 
Average number of teachers 41.4 29.2 45.7 71.2 25.3 22.7 18.2 27.3 31.0 
Average teacher-to-student ratio 1to 14 1to 13 1to 13 1to15 1to17 1to 18 1to 19 1to 14 1 to 16 
Eligible for Title | funds (percent) 76 88 70 50 71 75 61 76 56 
Located in a “Big Five” school 6 7 10 0 14 17 53 7 11 


district? (percent) 


Average demographic makeup (percent of students) 


Black 8 9 7 6 9 10 36 5 8 
Hispanic 12 13 13 8 10 11 8 9 8 
Other race/ethnicity? 13 15 12 10 7 8 8 7 

White 67 63 68 76 73 71 48 80 78 
Eligible for the national school 46 51 48 32 43 45 58 39 39 


lunch program 


Locale (percent of schools) 


City 42 41 60 25 25 28 54 16 20 
Suburb 30 30 20 40 20 23 10 20 15 
Town 15 13 10 25 19 18 10 24 18 
Rural 14 16 10 10 36 31 25 40 46 


Note: The analytic sample consisted of 2,290 elementary, middle, and high schools. 

a. The “Big Five” school districts are the five largest school districts in Wisconsin. These districts also serve the majority of Black students in the state. 

b. Includes American Indian students, Asian students, Pacific Islander students, students who identify as two or more races/ethnicities, and students with unknown race/ethnicity. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center, the Wisconsin Department of Public Instruction, and the U.S. Department of Education Common Core of Data. 


Comparing average school outcomes 


To answer research question 4, the study team compared average academic and behavior measures before and 
after program participation (English language arts test scores, math test scores, closing gaps scores for English 
language arts, closing gaps scores for math, attendance rate, suspension rate, and expulsion rate) for schools that 
participated in the program and schools that did not participate. 


Overall, there were few meaningful differences in academic and behavior measures before and after program 
participation between schools that participated in the program and schools that did not participate. There were 
no meaningful differences in English language arts test scores (table C2) or math test scores (table C3) between 
schools that participated in the program and schools that did not participate. Schools that participated in the 
program had lower closing gaps scores for English language arts’ three years after participation compared with 
schools that did not participate in the program (29.1 versus 32.4; table C4), and schools that participated in the 
program had lower closing gaps scores for math one year after participation (29.0 versus 31.2), two years after 
participation (29.0 compared with 31.6), and three years after participation (28.5 versus 32.1; table C5). Although 
the differences in closing gaps scores were meaningful, they were substantively small. With one exception, there 
were no meaningful differences in behavior outcomes, including attendance rate (table C6), suspension rate (table 
C7), and expulsion rate (table C8). The exception was that schools that participated in the program had higher 
suspension rates one year later (see table C7). For example, schools that participated in the program had a 
suspension rate of 4 percent one year after participation compared with 3 percent in schools that did not 
participate. These results are based on a small number of schools that participated in the program; therefore, the 
results may not be representative of all schools in Wisconsin. 


Table C2. Mean differences in English language arts test scores before participation and one year, two years, 
and three years after participation for schools that participated in the professional development program on 
culturally responsive practices in Wisconsin and for schools that did not participate, 2013/14-2018/19 


Number of English language arts Standard Effect 
participation status schools Fa oo) i=) deviation size 
Before participation 
Participated in program 69 —0.074 0.27 .06 
Did not participate in program 1,476 —0.047 0.44 
One year after participation 
Participated in program 87 —0.095 0.35 12 
Did not participate in program 1,862 —0.040 0.46 
Two years after participation 
Participated in program 66 —0.028 0.37 .0O 
Did not participate in program 1,431 —0.027 0.46 
Three years after participation 
Participated in program 64 —0.079 0.37 10 
Did not participate in program 1,372 —0.029 0.47 


Note: The effect size is Hedges’ g, which is a measure of how much one group differs from another group. The number of schools in each group is different 
because schools participated in different years. Schools that participated in the first year 2013/14 did not have any pre-participation data and were therefore 
dropped from the analysis. There is a similar problem for the post-participation data. Schools that participated in 2016/17—2018/19 did not have outcome 
data for all three years after participation and were therefore dropped from the analysis for which data were not available. Other variations in the number 
of schools each year are related to levels of missing data on the outcome variable. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center and the Wisconsin Department of Public Instruction. 


4 The closing gaps score is a continuous variable on a scale from 1 to 100. A higher score means that the gap between target groups and 
comparison groups is narrowing. 
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Table C3. Mean differences in math test scores before participation and one year, two years, and three years 
after participation for schools that participated in the professional development program on culturally 
responsive practices in Wisconsin and for schools that did not participate, 2013/14—-2018/19 


Time period and program Number of Math Standard 

participation status schools z score deviation 

Before participation 

Participated in program 69 —0.045 0.31 —.06 
Did not participate in program 1,476 —0.072 0.46 

One year after participation 

Participated in program 87 —0.076 0.37 .04 
Did not participate in program 1,862 -0.055 0.49 

Two years after participation 

Participated in program 66 —0.020 0.41 —.04 
Did not participate in program 1,431 -0.039 0.48 

Three years after participation 

Participated in program 64 —0.040 0.39 .0O 
Did not participate in program 1,373 -0.039 0.49 


Note: The effect size is Hedges’ g, which is a measure of how much one group differs from another group. The number of schools in each group is different 
because schools participated in different years. Schools that participated in the first year 2013/14 did not have any pre-participation data and were therefore 
dropped from the analysis. There is a similar problem for the post-participation data. Schools that participated in 2016/17—2018/19 did not have outcome 
data for all three years after participation and were therefore dropped from the analysis for which data were not available. Other variations in the number 
of schools each year are related to levels of missing data on the outcome variable. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center and the Wisconsin Department of Public Instruction. 


Table C4. Mean differences in closing gaps score for English language arts before participation and one year, 
two years, and three years after participation for schools that participated in the professional development 
program on culturally responsive practices in Wisconsin and for schools that did not participate, 2013/14- 


Number of Closing gaps score for Standard 
schools English language arts deviation 

Before participation 
Participated in program 68 31.14 9.16 .04 
Did not participate in program 1,190 31.43 7.98 
One year after participation 
Participated in program 55 31.45 9.42 10 
Did not participate in program 973 32.40 9.48 
Two years after participation 
Participated in program 44 30.74 12.65 15 
Did not participate in program 760 32.42 11.05 
Three years after participation 
Participated in program 64 29.09 12.37 30 
Did not participate in program 1,126 32.44 11.06 


Note: The effect size is Hedges’ g, which is a measure of how much one group differs from another group. The number of schools in each group is different 
because schools participated in different years. Schools that participated in the first year 2013/14 did not have any pre-participation data and were therefore 
dropped from the analysis. There is a similar problem for the post-participation data. Schools that participated in 2016/17—2018/19 did not have outcome 
data for all three years after participation and were therefore dropped from the analysis for which data were not available. Other variations in the number 
of schools each year are related to levels of missing data on the outcome variable. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center and the Wisconsin Department of Public Instruction. 
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Table C5. Mean differences in closing gaps scores for math before participation and one year, two years, and 
three years after participation for schools that participated in the professional development program on 
culturally responsive practices in Wisconsin and for schools that did not participate, 2013/14—2018/19 


Time period and program Number of Closing gaps score Standard Effect 
participation status schools for math deviation size 
Before participation 

Participated in program 68 29.45 8.54 13 
Did not participate in program 1,189 30.41 7.62 

One year after participation 

Participated in program 55 29.01 10.24 31 
Did not participate in program 973 31.91 9.37 

Two years after participation 

Participated in program 44 28.98 10.99 25 
Did not participate in program 760 31.57 10.49 

Three years after participation 

Participated in program 64 28.54 10.00 34 
Did not participate in program 1,126 32.06 10.46 


Note: The effect size is Hedges’ g, which is a measure of how much one group differs from another group. The number of schools in each group is different 
because schools participated in different years. Schools that participated in the first year 2013/14 did not have any pre-participation data and were therefore 
dropped from the analysis. There is a similar problem for the post-participation data. Schools that participated in 2016/17—2018/19 did not have outcome 
data for all three years after participation and were therefore dropped from the analysis for which data were not available. Other variations in the number 
of schools each year are related to levels of missing data on the outcome variable. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center and the Wisconsin Department of Public Instruction. 


Table C6. Mean differences in attendance rates before participation and one year, two years, and three years 
after participation for schools that participated in the professional development program on culturally 
responsive practices in Wisconsin and for schools that did not participate, 2013/14-2018/19 


Time period and program Number of Attendance Standard Effect 
participation status schools rate deviation size 
Before participation 

Participated in program 72 .953 0.01 -.15 
Did not participate in program 1,597 .948 0.04 

One year after participation 

Participated in program 92 .949 0.02 —.07 
Did not participate in program 2,003 .946 0.05 

Two years after participation 

Participated in program 70 951 0.02 —-.09 
Did not participate in program 1,529 .947 0.06 

Three years after participation 

Participated in program 68 .947 0.02 —.04 
Did not participate in program 1,469 945 0.05 


Note: The effect size is Hedges’ g, which is a measure of how much one group differs from another group. The number of schools in each group is different 
because schools participated in different years. Schools that participated in the first year 2013/14 did not have any pre-participation data and were therefore 
dropped from the analysis. There is a similar problem for the post-participation data. Schools that participated in 2016/17—2018/19 did not have outcome 
data for all three years after participation and were therefore dropped from the analysis for which data were not available. Other variations in the number 
of schools each year are related to levels of missing data on the outcome variable. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center and the Wisconsin Department of Public Instruction. 
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Table C7. Mean differences in suspension rates before participation and one year, two years, and three years 
after participation for schools that participated in the professional development program on culturally 
responsive practices in Wisconsin and for schools that did not participate, 2013/14—2018/19 


Time period and program Number of Suspension Standard 

schools rate deviation 
Before participation 
Participated in program 72 0.039 0.04 -.13 
Did not participate in program 1,597 0.032 0.06 
One year after participation 
Participated in program 92 0.039 0.04 -.19 
Did not participate in program 2,007 0.029 0.05 
Two years after participation 
Participated in program 70 0.035 0.03 -.14 
Did not participate in program 1,553 0.028 0.05 
Three years after participation 
Participated in program 68 0.035 0.03 -.12 
Did not participate in program 1,469 0.028 0.05 


Note: The effect size is Hedges’ g, which is a measure of how much one group differs from another group. The number of schools in each group is different 
because schools participated in different years. Schools that participated in the first year 2013/14 did not have any pre-participation data and were therefore 
dropped from the analysis. There is a similar problem for the post-participation data. Schools that participated in 2016/17—2018/19 did not have outcome 
data for all three years after participation and were therefore dropped from the analysis for which data were not available. Other variations in the number 
of schools each year are related to levels of missing data on the outcome variable. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center and the Wisconsin Department of Public Instruction. 


Table C8. Mean differences in expulsion rates before participation and one year, two years, and three years 
after participation for schools that participated in the professional development program on culturally 
responsive practices in Wisconsin and for schools that did not participate, 2013/14—2018/19 


Time period and program Number of Expulsion 

participation status schools rate deviation 

Before participation 

Participated in program 72 0.0005 0.00 01 
Did not participate in program 1,597 0.0005 0.00 

One year after participation 

Participated in program 92 0.0003 0.00 .05 
Did not participate in program 2,007 0.0006 0.01 

Two years after participation 

Participated in program 70 0.0002 0.00 .06 
Did not participate in program 1,533 0.0003 0.00 

Three years after participation 

Participated in program 68 0.0003 0.00 .01 
Did not participate in program 1,469 0.0004 0.00 


Note: The effect size is Hedges’ g, which is a measure of how much one group differs from another group. The number of schools in each group is different 
because schools participated in different years. Schools that participated in the first year 2013/14 did not have any pre-participation data and were therefore 
dropped from the analysis. There is a similar problem for the post-participation data. Schools that participated in 2016/17—2018/19 did not have outcome 
data for all three years after participation and were therefore dropped from the analysis for which data were not available. Other variations in the number 
of schools each year are related to levels of missing data on the outcome variable. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center and the Wisconsin Department of Public Instruction. 
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Regression results 

The study team examined whether program participation was meaningfully related to school-level outcomes one 
year, two years, and three years later, after school characteristics and pre-program academic and behavior 
measures were accounted for. 


Although program participation and school outcomes were not meaningfully related, some meaningful 
differences were found between school-level characteristics and school outcomes. 


Several variables predicted closing gaps scores for English language arts (table C9). Schools with more students 
had lower closing gaps scores for English language arts three years after participation (but not one year or two 
years after participation). Schools with more teachers had lower closing gaps scores in English language arts two 
year after participation (but not one year or three years after participation). Schools with higher percentages of 
Black students had lower closing gaps scores for English language arts one year, two years, and three years after 
participation. Schools with higher percentages of White students had lower closing gaps scores for English 
language arts scores one year, two years, and three years after participation. Schools with higher closing gaps 
scores in English language arts before participation had higher closing gaps scores one year, two years, and three 
years after participation. 


A few variables also predicted closing gaps scores in math (table C10). Schools with more students had lower 
closing gaps scores in math one year and three years after participation (but not two years after participation). 
Schools with more teachers had lower closing gaps scores in math arts two year after participation. Schools with 
higher closing gaps scores in math before participation had higher closing gaps scores one year, two years, and 
three years after participation. 


Schools with higher percentages of students eligible for the national school lunch program had lower English 
language arts test scores three years after participation, and schools with higher pre-program English language 
arts and math scores had higher test scores one year, two years, and three years after participation (tables C11 
and C12). 


Only two variables were meaningfully related to attendance rates (table C13). Schools with higher attendance 
rates before participation had higher attendance rates one year, two years, and three years after participation, 
and schools with higher percentages of Black students had lower attendance rates three years after participation 
(but not one year or two years after participation). 


Average school enrollment, average number of teachers, and pre-program suspension rates were the only 
variables that predicted suspension rates (table C14). Schools with more students had higher suspension rates 
three years after participation (but not one year or two years after participation). Schools with more teachers had 
lower suspension rates three years after participation (but not one year or two years after participation). Schools 
with higher suspension rates before participation had higher suspension rates one year, two years, and three 
years after participation. 


The characteristics that predicted expulsion rates were the number of teachers, pre-program expulsion rates, and 
the percentage of students eligible for the national school lunch program (table C15). Schools with more teachers 
had lower expulsion rates one year after participation (but not two years or three years after participation). 
Schools with higher percentages of students eligible for the national school lunch program had higher expulsion 
rates one year after participation (but not two years or three years after participation). Schools with higher 
expulsion rates before participation had higher expulsion rates two years and three years after participation. 
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Table C9. The relationship between participation in the professional development program on culturally 
responsive practices in Wisconsin and closing gaps scores for English language arts one year, two years, and 
three years later, 2013/14-2018/19 


Closing gaps scores for Closing gaps scores for English Closing gaps scores for English 
English language arts one language arts two years after language arts three years after 
year after participation participation participation 
(n = 587) (n = 752) w=) 
Standardized Standardized Standardized 
Characteristic Coefficient coefficient Coefficient coefficient Coefficient coefficient 
Average school -0.01 —.15 —0.00 —.08 -0.01 -.20 
enrollment 
Number of teachers —0.03 —.06 —0.14 —.25 —0.07 —.12 
Teacher-to-student 0.02 01 0.08 .04 0.08 .02 
ratio 
Eligible for Title | 1.10 -.05 1.92 .07 2.67 11 
funds 
Percentage of Black —9.92 -.18 —13.50 —.24 —12.63 -.22 
students 
Percentage of White —9.15 —.23 —9.66 —.23 —13.03 ~.31 
students 
Percentage of —6.60 -.14 —5.68 -.11 -7.18 -.14 


students eligible for 
the national school 
lunch program 


Pre-program closing 0.73 57 0.49 34 0.48 31 
gaps scores for English 
language arts 


Program participation 0.57 .01 -0.10 —.00 -1.67 —.04 


Note: Standardized coefficients were calculated by dividing the standard deviation of the predictor by the standard deviation of the outcome variable and 
multiplying that number by the regression coefficient. Effect sizes and standardized coefficients with an absolute value greater than .15 were considered 
meaningful. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center, the Wisconsin Department of Public Instruction, and the U.S. 
Department of Education’s Common Core of Data. 


Table C10. The relationship between participation in the professional development program on culturally 
responsive practices in Wisconsin and closing gaps scores for math one year, two years, and three years later, 
2013/14-2018/19 


Closing gaps scores for math Closing gaps scores for math Closing gaps scores for math 
one year after participation two years after participation three years after participation 
(n = 587) (wey) ew) 
Standardized Standardized Standardized 
Characteristic Coefficient coefficient Coefficient coefficient Coefficient coefficient 
Average school 0.01 -.30 —0.00 -.05 -0.01 -—.30 
enrollment 
Number of teachers 0.06 11 0.12 —.23 0.01 —.00 
Teacher-to-student 0.10 .07 0.03 .02 0.13 .03 
ratio 
Eligible for Title | 0.39 —.02 1.03 .04 1.58 .07 
funds 
Percentage of Black -3.61 -.07 -1.75 -.03 —2.23 —.04 
students 
Percentage of White 3.81 -.10 —.86 -.02 —5.83 -.15 
students 
Percentage of —4.09 -.09 —4.65 -.10 -6.28 -.14 


students eligible for 
the national school 
lunch program 


Pre-program closing 0.57 46 0.44 32 0.41 31 
gaps scores for math 
Program participation —2.27 -.05 -0.75 —.02 —1.56 —.04 


Note: Standardized coefficients were calculated by dividing the standard deviation of the predictor by the standard deviation of the outcome variable and 
multiplying that number by the regression coefficient. Effect sizes and standardized coefficients with an absolute value greater than .15 were considered 
meaningful. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center, the Wisconsin Department of Public Instruction, and the 
U.S. Department of Education’s Common Core of Data. 


Table C11. The relationship between participation in the professional development program on culturally 
responsive practices in Wisconsin and English language arts test scores one year, two years, and three years 
later, 2013/14-2018/19 


English language arts test English language arts test English language arts test 
scores one year after scores two years after scores three years after 
rey-Tad (all ese} a ela) feeTad(olf ode} (e)) fek-Tad(oll of} ae) a) 
(n = 1,389) (n = 948) (ee -t:10) 
Standardized Standardized Standardized 

Characteristic Coefficient coefficient Coefficient coefficient Coefficient coefficient 
Average school enrollment 0.00 .06 0.00 12 —0.00 —.11 
Number of teachers —0.00 —.08 —0.00 -.15 —0.00 —.06 
Teacher-to-student ratio 0.00 01 0.00 -.02 —0.00 —.07 
Eligible for Title | funds —0.01 —.00 —0.02 —.02 —0.00 —.00 
Percentage of Black students —0.08 -.03 —0.27 -.11 —0.30 -.14 
Percentage of White students —0.02 -.01 -0.01 -.01 -0.77 —.04 
Percentage of students eligible -0.23 -.11 —0.12 —.06 —0.40 -.17 
for the national school lunch 
program 
Pre-program English language 0.89 79 0.88 76 0.81 .68 
arts test scores 
Program participation -0.01 -.01 .00 .0O -0.04 —.02 


Note: Standardized coefficients were calculated by dividing the standard deviation of the predictor variable by the standard deviation of the outcome variable 
and multiplying that number by the regression coefficient. Effect sizes and standardized coefficients with an absolute value greater than .15 were considered 
meaningful. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center, the Wisconsin Department of Public Instruction, and the U.S. 
Department of Education’s Common Core of Data. 


Table C12. The relationship between participation in the professional development program on culturally 
responsive practices in Wisconsin and math test scores one year, two years, and three years later,2013/14- 
2018/19 


Math test scores one year Math test scores two years after Math test scores three years 
after participation fe-Tad(a] eye} ale) a) after participation 
(n = 1,389) (n = 948) (n = 887) 
Standardized Standardized Standardized 

Characteristic Coefficient coefficient Coefficient coefficient Coefficient coefficient 
Average school 0.00 .04 0.00 10 —0.00 —.08 
enrollment 
Number of teachers —0.00 -.05 —0.00 —.12 0.00 -.05 
Teacher-to-student 0.00 .05 0.00 01 0.01 .08 
ratio 
Eligible for Title | —0.00 —.00 —0.02 —.02 0.00 .0O 
funds 
Percentage of Black —0.02 -.01 0.16 —.06 0.12 -.05 
students 
Percentage of White 0.09 05 0.15 .07 0.18 10 
students 
Percentage of -0.18 -.08 -0.19 -.09 0.31 —.14 


students eligible for 
the national school 
lunch program 


Pre-program math 0.88 .80 0.79 72 0.75 .80 
test scores 


Program participation —0.02 -.01 0.02 .01 —0.00 —.00 


Note: Standardized coefficients were calculated by dividing the standard deviation of the predictor variable by the standard deviation of the outcome variable 
and multiplying that number by the regression coefficient. Effect sizes and standardized coefficients with an absolute value greater than .15 were considered 
meaningful. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center, the Wisconsin Department of Public Instruction, and the U.S. 
Department of Education’s Common Core of Data. 

U.S. Department of Education’s Common Core of Data. 


Table C13. The relationship between participation in the professional development program on culturally 
responsive practices in Wisconsin and attendance rates one year, two years, and three years later, 2013/14— 
2018/19 


Attendance rates one year § Attendance ratestwo years Attendance rates three years 


after participation after participation after participation 
(n= 1,494) (n = 1,016) (n = 951) 
Se Se cet 
Characteristic Coefficient coefficient Coefficient coefficient Coefficient coefficient 
Average school enrollment —0.00 -.07 0.00 .07 —0.00 —.12 
Number of teachers 0.00 .06 —0.00 —.06 0.00 .04 
Teacher-to-student ratio —0.00 —.02 —0.00 —.03 —0.00 -.03 
Eligible for Title | funds —0.00 —.02 —0.00 .00 0.00 .04 
Percentage of Black students —0.02 -.09 —0.03 -.14 —0.04 -.19 
Percentage of White students —0.00 -.02 -0.01 -.05 0.00 .01 
Percentage of students —0.02 —.09 —0.00 —.00 —0.01 —.06 


eligible for the national 
school lunch program 


Pre-program attendance rate 0.90 .68 1.00 7 0.61 39 
Program participation 0.00 .0O 0.00 .0O 0.00 .01 


Note: Standardized coefficients were calculated by dividing the standard deviation of the predictor variable by the standard deviation of the outcome variable 
and multiplying that number by the regression coefficient. Effect sizes and standardized coefficients with an absolute value greater than .15 were considered 
meaningful. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center, the Wisconsin Department of Public Instruction, and the U.S. 
Department of Education’s Common Core of Data. 


Table C14. The relationship between participation in the professional development program on culturally 
responsive practices in Wisconsin and suspension rates one year, two years, and three years later, 2013/14— 
2018/19 


Suspension rates one year Suspension rates two years Suspension rates three years 
after participation after participation after participation 
(n = 1,497) (n = 1,018) (n = 951) 
Standardized Standardized Standardized 

Characteristic Coefficient coefficient Coefficient coefficient Coefficient coefficient 
Average school 0.00 .08 0.00 12 0.00 32 
enrollment 
Number of teachers —0.00 -.04 —0.00 —.06 —0.00 -.25 
Teacher-to-student ratio —0.00 -.03 —0.00 -.01 —0.00 —.00 
Eligible for Title | funds —0.00 -.00 0.00 .05 0.01 05 
Percentage of Black —0.00 -.01 0.04 13 0.03 13 
students 
Percentage of White —0.00 —.02 —-0.00 -.01 —-0.00 -.01 
students 
Percentage of students 0.02 .07 0.01 .05 0.02 10 


eligible for the national 
school lunch program 


Pre-program suspension 0.78 .80 0.63 .65 0.57 .61 
rate 
Program participation 0.00 .01 0.00 .01 0.00 .01 


Note: Standardized coefficients were calculated by dividing the standard deviation of the predictor variable by the standard deviation of the outcome variable 
and multiplying that number by the regression coefficient. Effect sizes and standardized coefficients with an absolute value greater than .15 were considered 
meaningful. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center, the Wisconsin Department of Public Instruction, and the U.S. 
Department of Education’s Common Core of Data. 


Table C15. The relationship between participation in the professional development program on culturally 
responsive practices in Wisconsin and expulsion rates one year, two years, and three years later, 2013/14— 
2018/19 


Expulsion rates one year Expulsion rates two years Expulsion rates three years 
after participation after participation after participation 
(n= 1,497) (n = 1,018) (n= 951) 
Standardized Standardized Standardized 
Characteristic Coefficient coefficient Coefficient coefficient Coefficient coefficient 
Average school enrollment 0.00 .0O —0.00 -.11 —0.00 —.04 
Number of teachers —0.00 -.21 0.00 13 0.00 .04 
Teacher-to-student ratio —0.00 -.05 —0.00 —.02 —0.00 —.05 
Eligible for Title | funds —0.01 -.09 0.00 .02 —0.00 —.00 
Percentage of Black students —0.00 -.01 0.00 .05 0.00 01 
Percentage of White students 0.00 13 —0.00 —.00 —0.00 —.07 
Percentage of students eligible 0.00 19 —0.00 —.01 0.00 .09 
for the national school lunch 
program 
Pre-program expulsion rate 0.27 .07 0.35 30 0.24 .20 
Program participation 0.00 —.00 0.00 .02 —0.00 -.01 


Note: Standardized coefficients were calculated by dividing the standard deviation of the variable by the standard deviation of the outcome variable and 
multiplying that number by the regression coefficient. Effect sizes and standardized coefficients with an absolute value greater than .15 were considered 
meaningful. 

Source: Authors’ analysis of data from the Wisconsin Response to Intervention Center, the Wisconsin Department of Public Instruction, and the U.S. 
Department of Education’s Common Core of Data. 


